Portable display device, method of controlling portable display device, program, and recording medium

ABSTRACT

Provided is a portable display device sufficiently small to be held with one hand that enters a state for accepting a gesture when a fixed coordinate position near a central portion between a left display unit ( 14   a ) and a right display unit ( 14   b ) is pressed with a thumb Ftl of one hand holding the device, and accepts a command for performing such as page flipping processing based on a gesture inputted with an index finger Ffr of the other hand. Thus, it is possible to achieve an interface for input operations suitable for a two-screen display screen, where holding a two-screen portable display device naturally causes the device to enter a command accepting state to allow gesture recognition, and to enter a command non-accepting state when the portable display device is not held, in order to prevent a command from being falsely executed due to an unintended contact and such to the display screen.

REFERENCE TO RELATED APPLICATIONS

This application is a national stage application under 35 USC 371 ofInternational Application No. PCT/JP2010/054808, filed Mar. 19, 2010,which claims the priority of Japanese Application No. JP2009-177320,filed Jul. 30, 2009, the contents of which prior applications areincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to portable display devices, and inparticular to a portable display device having a two-screen display unitand a touch panel capable of detecting coordinates on the display unitthat are subjected to approach, contact, or press.

BACKGROUND OF THE INVENTION

In recent years, an increasing number of portable display devicesrequiring operations such as menu selection are provided with a touchpanel with which an operation such as selection of a desired menu isaccepted by pressing the panel using a stylus pen or a finger accordingto the display in the screen. Such portable display devices employvarious well-known methods, such as a resistive type, an electrostaticcapacitance type, an optical sensor type, and an infrared type, in orderto specify a position that has been pressed of the panel.

Further, in recent years, a display screen of such portable displaydevices is configured by two screens (typically, center spread), as itis preferable that the display screen account for a larger portion of ahousing surface. Moreover, some displays are configured such that twoscreens are connected without a seam (seamlessly).

Japanese Patent Application Laid-open No. 2006-53678 discloses astructure of a notebook computer having such a two-screen touch panel,and a configuration of a user interface, such as a virtual keyboard anda virtual mouse, on a display screen of the device.

PRIOR ART DOCUMENT Patent Document

-   [Patent Document 1] Japanese Patent Application Laid-open No.    2006-53678

SUMMARY OF THE INVENTION

In this case, among the conventional portable display devices asdescribed above, a notebook computer and such are assumed to be usedwhile being placed on such as a desk in use even though such a device isportable, and it is preferable to employ a configuration for such adevice in which an interface screen such as a virtual keyboard or avirtual mouse is displayed to accept an input.

However, in a case of a portable display device that is used with onehand, such as a mobile telephone terminal or a PDA (Personal DigitalAssistant) device, it is not necessarily suitable for such a device tobe operated through a virtual keyboard or a virtual mouse.

Further, in a case of a device or application software (also referred toas an electronic book) that is intended to use both of the two screensas a display screen, such as a device or application software forpresenting an electronic document in a two-page center spread mannerlike a printed book, displaying the virtual keyboard or the virtualmouse on the display screen interferes the view, and therefore a methodfor more suitably accepting an input operation is required.

Thus, an object of the present invention is to provide a portabledisplay device that is sufficiently small to be held with one hand andprovided with an interface for an input operation suitable for atwo-screen display screen, a method of controlling the portable displaydevice, a program, and a recording medium.

A first aspect of the present invention provides a portable displaydevice configured to be held by a user, the device including:

a display unit configured to display images respectively in two adjacentscreens;

an input unit configured to obtain two or more coordinates of positionson the display unit, the positions being subjected to one of approach,contact, and press by the user; and

a gesture recognizing unit configured to recognize and execute apreviously stored processing command corresponding to a changing form oftwo or more related coordinates by chronologically associating thecoordinates of the positions subjected to one of approach, contact, andpress on the display unit and obtained in the input step, wherein

the gesture recognizing unit enters: a non-accepting state in whichrecognition and execution of the processing command are unallowed whenthe coordinates obtained in the input step do not include apredetermined fixed coordinate, and an accepting state in whichrecognition of the processing command is allowed when the coordinatesobtained by the input unit include the fixed coordinate.

According to a second aspect of the present invention, in the firstaspect of the present invention,

the gesture recognizing unit executes the processing command based onthe fixed coordinate that have been previously determined, the fixedcoordinate being at or near a portion at which the two screens areadjacent.

According to a third aspect of the present invention, in the firstaspect of the present invention,

the input unit obtains, as the fixed coordinate, a coordinate of aposition on the display unit that is corresponding to a position of onehand finger of the user holding the portable display device and that iscorresponding to a position suitable for being held by the one handfinger of the user, and obtains, as the two or more related coordinates,coordinates of positions on the display unit that are corresponding to aposition of the other hand finger of the user.

According to a fourth aspect of the present invention, in the firstaspect of the present invention,

the display unit is foldable such that the two screens face toward eachother taking a portion at or near a side where the two screens areadjacent as a valley-folding line, and

the gesture recognizing unit executes the processing command based onthe fixed coordinate including a coordinate on or near thevalley-folding line.

According to a fifth aspect of the present invention, in the firstaspect of the present invention,

when the two or more related coordinates represent a changing form inwhich the coordinates move from one to the other of the two screens, thegesture recognizing unit recognizes and executes a predetermined pageforwarding command corresponding to the changing form, and

the display unit displays two images respectively associated withpredetermined pages in a two-page center spread manner in the twoscreens, and displays images associated with pages incremented ordecremented by a predetermined number according to the changing formwhen the page forwarding command is executed by the gesture recognizingunit.

According to a sixth aspect of the present invention, in the firstaspect of the present invention,

when the coordinates obtained by the input unit include previouslydetermined coordinates successively for a predetermined time period, thegesture recognizing unit recognizes and executes a predetermined pageforwarding command during this period, the previously determinedcoordinates being positioned near an outer circumference of the twoscreens and distant from the fixed coordinate by a predetermineddistance, and

the display unit displays two images respectively associated withpredetermined pages in a two-page center spread manner in the twoscreens, and displays images associated with pages incremented ordecremented by a predetermined number during the period at apredetermined time interval when the page forwarding command isrecognized by the gesture recognizing unit.

According to a seventh aspect of the present invention, in the firstaspect of the present invention,

the display unit displays two images respectively associated withpredetermined pages in a two-page center spread manner in the twoscreens, and

when the two or more related coordinates represent a changing form inwhich the two or more related coordinates move from a portion near anend of the screen to a portion near a center of the screen at or near aportion at which the two screens are adjacent, the gesture recognizingunit recognizes a predetermined bookmark assigning command, and storespages corresponding to the images displayed in the display unit.

According to an eighth aspect of the present invention, in the firstaspect of the present invention,

when the two or more related coordinates represent a changing form inwhich the two or more related coordinates move from a portion near oneside to a portion near another side out of four outer sides of at leastone square display window in the two screens, the gesture recognizingunit recognizes and executes a predetermined window form changingcommand corresponding to the changing form, and

when the window form changing command is executed by the gesturerecognizing unit, the display unit changes one of the number and a formof divisions of the window according to the changing form, and displaysthe images in the window in the form after the change.

According to a ninth aspect of the present invention, in the firstaspect of the present invention,

the gesture recognizing unit enters: a non-accepting state in whichrecognition and execution of the processing command are unallowed whenthe coordinates obtained by the input unit do not include apredetermined number of fixed coordinates out of a plurality ofpredetermined fixed coordinates at or near a portion at which the twoscreens are adjacent, and an accepting state in which recognition of theprocessing command is allowed when the coordinates obtained by the inputunit include the predetermined number of fixed coordinates.

According to a tenth aspect of the present invention, in the firstaspect of the present invention,

during a time period in which the gesture recognizing unit is in thenon-accepting state, the input unit performs at least one of anoperation of limiting a range of coordinates to be obtained on thedisplay unit to the fixed coordinate or near the fixed coordinate, andan operation of setting a time interval at which the coordinates on thedisplay unit are to be obtained to be longer than a time period duringwhich the gesture recognizing unit is in the accepting state, therebyobtaining the coordinates.

An eleventh aspect of the present invention provides a method ofcontrolling a portable display device configured to be held by a userand having a display unit for displaying an image over two adjacentscreens, the method including:

an input step of obtaining two or more coordinates of positions on thedisplay unit, the positions being subjected to one of approach, contact,and press by the user; and

a gesture recognizing step of recognizing and executing a previouslystored processing command corresponding to a changing form of two ormore related coordinates by chronologically associating the coordinatesof the positions subjected to one of approach, contact, and press on thedisplay unit and obtained in the input step, wherein

the gesture recognizing step is in: a non-accepting state in whichrecognition and execution of the processing command are unallowed whenthe coordinates obtained in the input step do not include a previouslydetermined fixed coordinate at or near a portion at which the twoscreens are adjacent, and an accepting state in which recognition of theprocessing command is allowed when the coordinates obtained in the inputstep include the fixed coordinate.

A twelfth aspect of the present invention provides a program for causinga portable display device having a display unit for displaying an imageover two adjacent screens and configured to be held by a user toexecute:

an input step of obtaining two or more coordinates of positions on thedisplay unit, the positions being subjected to one of approach, contact,and press by the user; and

a gesture recognizing step of recognizing and executing a previouslystored processing command corresponding to a changing form of two ormore related coordinates by chronologically associating the coordinatesof the positions subjected to one of approach, contact, and press on thedisplay unit and obtained in the input step, wherein

the gesture recognizing step is in: a non-accepting state in whichrecognition and execution of the processing command are unallowed whenthe coordinates obtained in the input step do not include a previouslydetermined fixed coordinate at or near a portion at which the twoscreens are adjacent, and an accepting state in which recognition of theprocessing command is allowed when the coordinates obtained in the inputstep include the fixed coordinate.

A thirteenth aspect of the present invention provides acomputer-readable non-transitory recording medium having a programrecorded therein, the program causing a portable display device having adisplay unit for displaying an image over two adjacent screens andconfigured to be held by a user to execute:

an input step of obtaining two or more coordinates of positions on thedisplay unit, the positions being subjected to one of approach, contact,and press by the user; and

a gesture recognizing step of recognizing and executing a previouslystored processing command corresponding to a changing form of two ormore related coordinates by chronologically associating the coordinatesof the positions subjected to one of approach, contact, and press on thedisplay unit and obtained in the input step, wherein

the gesture recognizing step is in: a non-accepting state in whichrecognition and execution of the processing command are unallowed whenthe coordinates obtained in the input step do not include a previouslydetermined fixed coordinate at or near a portion at which the twoscreens are adjacent, and an accepting state in which recognition of theprocessing command is allowed when the coordinates obtained in the inputstep include the fixed coordinate.

According to the first aspect of the present invention, the deviceenters: a non-accepting state in which recognition and execution of theprocessing command are unallowed when the coordinates obtained by theinput unit do not include a predetermined fixed coordinate, and anaccepting state in which recognition of the processing command isallowed when the coordinates obtained by the input unit include thefixed coordinate. Accordingly, pressing a fixed coordinate position witha thumb of one hand holding the device, for example, causes the portabledisplay device to enter a state in which the gesture recognition can beaccepted. Therefore, it is possible to conveniently cause the portabledisplay device to enter the command accepting state when holding thedevice to allow the gesture recognition, and to enter the commandnon-accepting state typically when not holding the device, to prevent acommand from being falsely executed due to an unintended contact andsuch to the display screen.

According to the second aspect of the present invention, pressing apreviously determined fixed coordinate position at or near the portionat which the two screens are adjacent with a thumb of one hand holdingthe device, for example, causes the device to enter a state foraccepting the gesture recognition. Therefore, it is possible tonaturally cause the device to enter the command accepting state to allowthe gesture recognition when holding the two-screen portable displaydevice, and to enter the command non-accepting state when not holdingthe device to prevent a command from being falsely executed. In thismanner, it is possible to achieve an interface for an input operationsuitable for a two-screen display screen.

According to the third aspect of the present invention, the input unitobtains, as the fixed coordinate, a coordinate of positions that iscorresponding to a position of one hand finger of the user holding theportable display device and that is corresponding to a position suitablefor being held, and obtains, as the two or more related coordinates,coordinates of positions on the display unit that are corresponding to aposition of the other hand finger of the user. Therefore, it is possibleto conveniently cause the device to enter the command accepting state toallow the gesture recognition when holding the portable display device,and to enter the command non-accepting state when not holding the deviceto prevent a command from being falsely executed.

According to the fourth aspect of the present invention, the processingcommand is executed based on the fixed coordinate including a coordinateon or near the valley-folding line. Therefore, it is expected that thedevice be held like a printed book, for example, by pressing the thumbto hold the device along the valley-folding line, and it is possible tomore naturally cause the device to enter the command accepting state toallow the gesture recognition, and to enter the command non-acceptingstate when not holding the device, to prevent a command from beingfalsely executed.

According to the fifth aspect of the present invention, when the two ormore related coordinates represent a changing form in which thecoordinates move from one to the other of the two screens, apredetermined page forwarding command corresponding to the changing formis recognized and executed by the gesture recognizing unit. Therefore,it is possible to facilitate the execution of the page forwarding by anintuitive gesture over two screens of flipping a page.

According to the sixth aspect of the present invention, when thecoordinates obtained by the input unit include previously determinedcoordinates successively for a predetermined time period, apredetermined page forwarding command is recognized and executed duringthis period, the previously determined coordinates being positioned nearan outer circumference of the two screens and distant from the fixedcoordinate by a predetermined distance. Therefore, it is possible tofacilitate the execution of the successive page forwarding by a simplemotion such as continuously pressing the coordinate position.

According to the seventh aspect of the present invention, when the twoor more related coordinates represent a changing form in which the twoor more related coordinates move from a portion near an end of thescreen to a portion near a center of the screen at or near a portion atwhich the two screens are adjacent, a predetermined bookmark assigningcommand corresponding to the changing form is recognized, and pagescorresponding to the images displayed in the display unit are stored.Therefore, it is possible to facilitate the execution of bookmarkapplication processing by a simple and intuitive gesture of applying abookmark.

According to the eighth aspect of the present invention, when the two ormore related coordinates represent a changing form in which the two ormore related coordinates move from a portion near one side to a portionnear another side out of four outer sides of at least one square displaywindow in the two screens, a predetermined window form changing commandcorresponding to the changing form is recognized and executed.Therefore, it is possible to easily change the number or the form ofdivisions of the window by a simple and intuitive gesture.

According to the ninth aspect of the present invention, the gesturerecognizing unit enters: a non-accepting state in which recognition andexecution of the processing command are unallowed when the coordinatesobtained by the input unit do not include a predetermined number offixed coordinates out of the plurality of fixed coordinates, and anaccepting state in which recognition of the processing command isallowed when the coordinates obtained by the input unit include thepredetermined number of fixed coordinates. Therefore, it is possible toprevent the false detection from occurring as compared to thedetermination based on a single coordinate.

According to the tenth aspect of the present invention, during a timeperiod in the non-accepting state, the input unit performs at least oneof an operation of limiting a range of coordinates to be obtained on thedisplay unit to the fixed coordinate or near the fixed coordinate, andan operation of setting a time interval at which the coordinates are tobe obtained to be long, thereby obtaining the coordinates. Therefore, anoperation, a signal and the like that are used to obtain a coordinatewithin a range other than the fixed coordinate can be eliminated and thenumber of times the coordinate per unit time is obtained can be reduced.Thus, it is possible to reduce power consumption during thenon-accepting state.

According to the eleventh aspect of the present invention, it ispossible to provide the same effect as that of the first aspectaccording to the present invention for a method of controlling aportable display device.

According to the twelfth aspect of the present invention, it is possibleto provide the same effect as that of the first aspect according to thepresent invention for a program for a portable display device.

According to the thirteenth aspect of the present invention, it ispossible to provide the same effect as that of the first aspectaccording to the present invention for a computer-readablenon-transitory recording medium having a program for a portable displaydevice recorded therein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external perspective view illustrating a portableinformation terminal according to one embodiment of the presentinvention.

FIG. 2 is a block diagram illustrating a main configuration of a displayunit and an input unit of the portable information terminal according tothe embodiment.

FIG. 3 is a block diagram illustrating a configuration of the portableinformation terminal according to the embodiment.

FIG. 4 is a flowchart showing an overall processing flow of the portableinformation terminal according to the embodiment.

FIG. 5 is a flowchart showing a detailed flow of input processing (stepS2) according to the embodiment.

FIG. 6 is a view showing a positional relation between a display screenof the portable information terminal according to the embodiment and aleft thumb of a user.

FIG. 7 is a view showing a fixed coordinate area in the embodiment.

FIG. 8 is a flowchart showing a detailed flow of recognition processing(step S3) according to the embodiment.

FIG. 9 is a flowchart showing a detailed flow of book-reading processing(step S33) according to the embodiment.

FIG. 10 is a view showing a predetermined region at an upper center ofthe display unit according to the embodiment.

FIG. 11 is a view showing a predetermined region at a lower center ofthe display unit according to the embodiment.

FIG. 12 is a view showing an example of the screen shown when performingbookmark application processing according to the embodiment.

FIG. 13 is a view showing an example according to the embodiment, inwhich a gesture of a right index finger moving from a left screen to aright screen is inputted.

FIG. 14 is a view showing an example according to the embodiment, inwhich a gesture of the right index finger moving from the right screento the left screen is inputted.

FIG. 15 is a view showing an example according to the embodiment, inwhich a gesture of a right thumb pressing near a right end portion inthe right screen is inputted.

FIG. 16 is a flowchart showing a detailed flow of window processing(step S35) according to the embodiment.

FIG. 17 is a view showing a relation between changing forms of a windowand previously determined gestures according to the embodiment.

DETAILED DESCRIPTION OF THE INVENTION 1. Overall Device Configurationand Operation

FIG. 1 is an external perspective view illustrating a portableinformation terminal according to one embodiment of the presentinvention. A portable information terminal 10 is provided with a leftdisplay unit 14 a and a right display unit 14 b. The left display unit14 a and the right display unit 14 b are configured such that the leftdisplay unit 14 a and the right display unit 14 b are foldable facingtoward each other at central portion via a hinge unit that is notdepicted. FIG. 1 illustrates the portable information terminal slightlyfolded at the central portion of the device that is near a side wherethe left display unit 14 a and the right display unit 14 b are adjacent,showing a size, a shape, and a weight balance that can be held with onehand like a book spread in the center.

More specifically, the portable information terminal 10 is held with onehand of a user by, as will be described later, holding a portion arounda lower part of a center portion of the device with a thumb (typically,of a nondominant hand of the user), and the other side of the portionaround the lower part with a different finger. The portable informationterminal 10 has the shape and the weight balance that are suitable to beheld with one hand in this manner.

Further, although not shown in FIG. 1, a well-known light guiding member(for example, a prism or a light guiding element configured by a bundleof single fibers whose diameter is a few micrometers called a fiberplate) is provided at a portion between a right side portion of the leftdisplay unit 14 a and a left side portion of the right display unit 14 bthat corresponds to the central portion. By this light guiding member, apart of two images displayed in the left display unit 14 a and the rightdisplay unit 14 b is subjected to an optical path conversion, and asingle image is displayed without a cut line or a seam (seamlessly).Further, there is provided a transparent touch panel over an uppersurface (top surface) of the left display unit 14 a and the rightdisplay unit 14 b, and by pressing (contact) a screen with a finger(typically, of a dominant hand of the user) or a stylus pen, a positionpressed on the screen is detected. Configurations of the display unitand the touch panel will be described later.

FIG. 2 is a block diagram illustrating a main configuration of thedisplay unit and an input unit of the portable information terminalaccording to one embodiment of the present invention. The portableinformation terminal 10 is provided with a control unit 100, two liquidcrystal panels 141 respectively having the left display unit 14 a andthe right display unit 14 b, scanning drivers 142 and data drivers 143pairs of which respectively driving the two liquid crystal panels 141, adisplay control circuit 145, a matrix type resistive touch panel 161provided over the two liquid crystal panels 141, an X coordinate sensor163 and a Y coordinate sensor 162 for detecting a position on the touchpanel 161 that is pressed by the user's finger or a stylus pen, and acoordinate processing unit 165.

Here, the touch panel 161 is not a general resistive type touch panelthat senses a contact point of two resistance films that are facing eachother in an analog manner, but provided with a large number oftransparent electrodes parallelly provided in rows, and a large numberof transparent electrodes parallelly provided in columns in a directionperpendicular to the transparent electrodes so as to face toward thetransparent electrodes with a predetermined short distance therebetween.The X coordinate sensor 163 is connected to the electrodes provided incolumns, and the Y coordinate sensor 162 is connected to the electrodesprovided in rows. Therefore, when the electrodes in rows and in columnsthat intersect are brought into contact with each other at a positionpressed by such as the user's finger or a stylus pen, the position canbe detected by the X coordinate sensor 163 and the Y coordinate sensor162. Thus, a large number of coordinates on the touch panel 161 can beindividually recognized in a resolution according to an arrangementpitch of the electrodes.

Further, it is possible to employ various well-known types of touchpanels, such as a matrix-type electrostatic capacitance type, an opticalsensor type, and a mechanical sensor type, as long as the touch panel iscapable of individually recognizing a large number of coordinates.Moreover, a plurality of touch panels of a type that can recognize onlya single coordinate can be used in combination. For example, it ispossible to separately provide a touch panel corresponding to the leftdisplay unit 14 a, and a touch panel corresponding to the right displayunit 14 b, and a touch panel corresponding to a portion near a portionadjacent to these touch panels, which will be later described. In thismanner, it is possible to reduce manufacturing costs. It should be notedthat the touch panels of the electrostatic capacitance type and theoptical sensor type are often preferred, as these types of touch panelsdo not generally require to be pressed by the finger, and only lightlytouching the touch panel with the finger or causing the finger toapproach the touch panel is sufficient, unlike the touch panel of theresistive type.

Further, the two liquid crystal panels 141 are active matrix-type liquidcrystal panels, in which the scanning driver 142 and the data driver 143for each liquid crystal panel performs selection of pixels and dataassignment in the corresponding liquid crystal panel, thereby forming asingle image. It should be noted that in order to display the singleimage in the two liquid crystal panels 141, and to uniquely associatethe single image with detection coordinates in the touch panel 161,pixel positions on the liquid crystal panel 141 having the right displayunit 14 b can be treated as values obtained by adding a maximum value ofan X coordinate (a pixel position at a left end) to pixel positions onthe liquid crystal panels 141 having the left display unit 14 a.Moreover, as described above, in order to display as a single image, thelight guiding member is provided between the two liquid crystal panels141 to perform seamless display.

FIG. 3 is a block diagram illustrating a configuration of the portableinformation terminal according to the one embodiment of the presentinvention. The portable information terminal 10 is a device thatperforms predetermined processing based on a general (dedicated)operating system and predetermined application software, and providedwith the control unit 100 having a CPU (Central Processing Unit) and asemiconductor memory such as a RAM, a storage unit 120 having anonvolatile semiconductor memory such as an EPROM, a display unit 140configured by such as a liquid crystal panel that will be laterdescribed, and an input unit 160 having an input operation device suchas a touch panel.

The control unit 100 included in the portable information terminal 10has a function of predetermined command processing performed byrecognizing a gesture (described later) of the user accepted through theinput unit 160. An operation of the control unit 100 will be describedin detail later.

It should be noted that the function of the control unit 100 is achievedby the CPU executing a predetermined gesture recognizing program P (forexample, application software for gesture recognition) stored in thesemiconductor memory. Here, while the gesture recognizing program P iswritten in the EPROM and such at the time of manufacturing, the programcan be written after the time of manufacturing, for example, through aCD-ROM or a different type of non-transitory recording medium as acomputer-readable recording medium in which the program is recorded, orthrough a communication line. Then, upon performing a predeterminedoperation for activating the portable information terminal 10, a part orall of the gesture recognizing program P written in the storage unit 120is transferred to the semiconductor memory such as the RAM to betemporary stored therein, and executed by the CPU in the control unit100. This realizes processing to control the components of the controlunit 100.

2. Overall Operation of Portable Information Terminal

Next, an overall operation of the portable information terminal 10 isdescribed. FIG. 4 is a flowchart showing an overall processing flow ofthe portable information terminal 10. In step S1 (initial settingprocessing) shown in FIG. 4, the control unit 100 included in theportable information terminal 10 determines image data corresponding toan electronic document to be presented to the user, for example,typically by receiving a start instruction from the user. Further, thecontrol unit 100 initializes values necessary for the processing thatwill be later described. In this case, the number of windows wheninitialized is 1 (the window is not divided).

Here, while various well-known application software can be installed inthe portable information terminal 10, an explanation is given only to areading mode and a window mode. The reading mode is a mode for browsingelectric book data stored in the storage unit 120. The book data isimage data including character data, and stored in association with apredetermined page like a printed book. Further, the window mode is amode for instructing the number of windows and a form of division so asto display a plurality of application within one, two, or four windows.Here, for convenience of explanation, it is assumed that only processingcorresponding to the two modes are performed.

Next, in step S2 (input processing), the image determined in step S1 isdisplayed in the display unit 140, and an input operation from the inputunit 160 made by the user, in this case, an input operation by bringingthe finger into contact, is accepted.

In step S3 (recognition processing), according to the input operationaccepted in step S2, corresponding processing command is recognized, andan image corresponding to the recognized processing command is displayedin the display unit 140.

In step S4, an instruction to stop the device is given by the user, orit is determined whether or not the various processing is to beterminated by such as sleeping processing due to a lapse of apredetermined time. When the process is not terminated, the processreturns to step S2 and is repeated (S4→S2→S3→S4). When the process isterminated, the portable information terminal 10 terminates the processonce, and resumes the process typically when an instruction to activatethe device is given by the user.

3. Input Processing Operation of Portable Information Terminal

Next, an operation in the input processing (step S2) of the portableinformation terminal 10 is described in detail. FIG. 5 is a flowchartshowing a detailed flow of the input processing (step S2).

In step S21 shown in FIG. 5, the control unit 100 determines whether ornot the finger is placed on a fixed coordinate corresponding to apreviously determined position. Specifically, the control unit 100compares and determines whether a group of coordinates accepted from theinput unit 160 includes a coordinate within a fixed coordinate area. Inthe following description, the fixed coordinate area is described withreference to FIG. 6 and FIG. 7.

FIG. 6 is a view showing a positional relation between a display screenof the portable information terminal and a left thumb of the user, andFIG. 7 is a view showing the fixed coordinate area in provided near theleft thumb. As described above, the portable information terminal 10 isnaturally held with one hand of the user by holding the portion aroundthe lower part of the center portion of the terminal with a thumb (here,of a left hand, for convenience of explanation) of a nondominant hand ofthe user, and the other side of the portion around the lower part with adifferent finger. FIG. 6 shows a left thumb Ftl. It should beappreciated that it is possible to hold the terminal using such as adominant hand or an artificial hand.

Further, as shown in FIG. 7, a fixed coordinate area 16 is provided at alower part of an adjacent side between the left display unit 14 a andthe right display unit 14 b. The fixed coordinate area 16 includes aplurality of detection points Ps with which coordinates of positionspressed by the input unit 160 can be detected. Among these, thedetection points Ps that correspond to coordinates of positions that areactually being pressed are indicated by shaded circles. The coordinatescorresponding to the shaded circles are a part or all of the group ofthe coordinates accepted from the input unit 160, and the control unit100 determines whether or not any of the coordinates in the groupoverlaps with one or more (in order to avoid false detection, no smallerthan a predetermined number that is two or more) of the coordinateswithin the fixed coordinate area. It should be noted that the abovedescribed method of determining whether or not the fixed coordinate areaincludes the coordinates corresponding to the position of the left thumbis a mere example, and any well-known method can be used for thedetermination.

In step S21, the control unit 100 determines whether or not the fingeris placed on the fixed coordinate as described above. If it isdetermined that the finger has been placed (Yes in step S21), thecontrol unit 100 enters a state in which a command can be accepted(hereinafter also referred to as a “command accepting state”) that willbe later described, and the process proceeds to step S23. Further, if itis determined that the finger has not been placed (No in step S21), thecontrol unit 100 enters a state in which a command can not be accepted(hereinafter also referred to as a “command non-accepting state”) thatwill be later described, and the input processing is terminated and theprocessing shown in FIG. 4 is resumed. It should be noted that it isalso possible to employ a configuration in which a gesture other than agesture that will be described later is recognized, and in which commandprocessing corresponding to the recognized gesture (other than a commandthat will be later described) is performed.

Moreover, in the command non-accepting state, it is not necessary toperform related processing that should be performed in the commandaccepting state. Therefore, it is desirable that the sensors be drivenand the data be processed such that power consumption for driving thesensors and for processing the data is reduced, such as by loweringdrive frequencies of the X coordinate sensor 163 and the Y coordinatesensor 162 (sensor data readout frequencies) for performing thedetection in the touch panel 161 (for example, performing the detectionevery 60 frames), by lowering a drive frequency of a light source whenusing an optical sensor, or by preventing the sensor data readout in anarea other than the fixed coordinate area from being performed and thedate from being processed by the coordinate processing unit 165 b. Itshould be noted that when the state shifts to the command acceptingstate, the driving and the processing described above are resumed to anormal state.

Furthermore, it is also possible to employ a configuration in which acoordinate in the area other than the fixed coordinate area is detectedby the touch panel 161, a coordinate in the fixed coordinate area isdetected by a single-electrode resistive type (single) touch sensor or amechanical sensor that is different from the touch panel 161, and theoperation of the touch panel 161 may be completely terminated only inthe command non-accepting state. This allows reduction of the powerconsumption in the command non-accepting state.

Next, in step S23, the control unit 100 determines whether or not afinger is placed on a portion other than the fixed coordinate area 16.Specifically, the control unit 100 compares and determines whether thegroup of the coordinates accepted from the input unit 160 includes acoordinate outside the fixed coordinate area 16. As a result of thedetermination, if it is determined that a different finger has beenplaced (Yes in step S23), the process proceeds to step S25.Alternatively, if it is determined that the different finger has notbeen placed (No in step S23), the control unit 100 initializes astarting coordinate and an ending coordinate (step S24), and the inputprocessing is terminated and the processing shown in FIG. 4 is resumed.It should be noted that in the above determination, in order to avoidfalse detection as has been described above, it is preferable to compareand determine whether or not no smaller than two or a predeterminednumber of coordinates in the area other than the fixed coordinate area16 are included.

Subsequently, in step S25, the control unit 100 stores a coordinateoutside the fixed coordinate area 16 among the group of the coordinatesaccepted from the input unit 160 (when there are more than one,arbitrary one or an average coordinate of a group of contiguouscoordinates) as a starting coordinate (X1, Y1) in the storage unit 120,and starts measuring elapsed time that is required for gesturerecognition that will be described later. It should be noted that whilethe gesture normally means a form how a coordinate changeschronologically as the user's finger moves over the display unit, thegesture herein includes a form in which the coordinate does not changefrom a predetermined coordinate for predetermined time.

Next, in step S27, the control unit 100 determines whether or not thefinger is lifted from the area other than the fixed coordinate area 16,and whether or not predetermined time has elapsed. Specifically, thecontrol unit 100 compares and determines whether or not there isestablished one of conditions that the group of the coordinates acceptedfrom the input unit 160 does not include the coordinate outside thefixed coordinate area 16 and that the predetermined time has elapsed. Asa result of the determination, if it is determined that the differentfinger has been lifted or the predetermined time has elapsed (Yes instep S27), the process proceeds to step S29. Alternatively, if it isdetermined that the different finger has not been lifted and thepredetermined time has not yet elapsed (No in step S25), this processing(S27) is repeated until it is determined that the different finger islifted or the predetermined time has elapsed.

Although the predetermined time is set to be long enough so as not to betaken as a common gesture as time for general time-out processing, thepredetermined time here is set in association with the inputtedcoordinate. Specifically, if the inputted coordinate does not change(over a predetermined range), a short time (for example, on the order of1 second) is set, and the process proceeds to subsequent processing uponelapsing of this time. Alternatively, it is also possible to employ aconfiguration in which if the inputted coordinates are within thepredetermined coordinate range associated with a successive pageflipping operation that will be described later (specifically,predetermined ranges 33 a and 33 b described later and shown in FIG.15), the short time is set, and the process proceeds to subsequentprocessing upon elapsing of this time. In such a case, it is desirablethat the process proceeds to recognition processing (step S3) in a shorttime as the coordinate specification is possibly for the successive pageflipping operation that will be described later. Further, it is alsopossible to seta shorter time (for example, on the order of 0.2 seconds)when the predetermined time elapses more than one time, so that a speedof the successive page flipping operation consequently increases.

It should be noted that the repeating processing described above iscanceled by such as predetermined interrupt processing so that theprocess proceeds to processing in step S29. Further, in the abovedetermination, in order to avoid false detection as has been describedabove, it is preferable to compare and determine whether or notcoordinates in the area other than the fixed coordinate area 16 areincluded, and whether or not no smaller than two or a predeterminednumber of coordinates are included.

Subsequently, in step S29, the control unit 100 stores a coordinateoutside the fixed coordinate area 16 among the group of the coordinatesmost recently accepted from the input unit 160 (when there are more thanone, arbitrary one or an average coordinate of a group of contiguouscoordinates) as an ending coordinate (X2, Y2) in the storage unit 120,and the measurement of the elapsed time is terminated. Specifically, itis possible for the control unit 100 to store a coordinate immediatelybefore the finger is lifted as the ending coordinate by, in step S27,continuously storing the coordinate outside the fixed coordinate area 16among the group of the coordinates accepted from the input unit 160(when there are more than one, arbitrary one or an average coordinate ofa group of contiguous coordinates) as an ending coordinate (X2, Y2) inthe storage unit 120. Then, the input processing is terminated and theprocessing shown in FIG. 4 is resumed.

4. Recognition Processing Operation of Portable Information Terminal 4.1Overall Flow of Recognition Processing

Next, an operation in the recognition processing (step S3) of theportable information terminal 10 is described in detail. FIG. 8 is aflowchart showing a detailed flow of the recognition processing (stepS3).

In step S31 shown in FIG. 8, the control unit 100 determines whether ornot the reading mode is selected by the user. As a result of thedetermination, if it is determined to be the reading mode (Yes in stepS31), book-reading processing in step S33 is performed. Alternatively,if it is determined to be not the reading mode (No in step S31), windowprocessing in step S35 is performed. After the processing end, therecognition processing is terminated and the processing shown in FIG. 4is resumed.

As described above, for convenience of explanation, it should be notedthat only the two processing of the reading mode and the window mode areperformed here. However, various well-known processing can be performedin practice. Next, the book-reading processing in the reading mode isdescribed in detail.

4.2 Book-Reading Processing Operation in Reading Mode

FIG. 9 is a flowchart showing a detailed flow of the book-readingprocessing (step S33). In step S331 shown in FIG. 9, the control unit100 determines whether or not the starting coordinate (X1, Y1) obtainedin step S25 shown in FIG. 5 is within a predetermined region at an uppercenter of the display unit.

FIG. 10 is a view showing the predetermined region at the upper centerof the display unit. As shown in FIG. 10, when the upper center of thedisplay unit is pressed (typically) with a right index finger Ffr, acoordinate corresponding to the pressed position is stored as thestarting coordinate (X1, Y1). In this case, the right index finger Ffrpresses a portion within a predetermined region 31 in the upper center,and therefore it is determined that the starting coordinate (X1, Y1) iswithin the predetermined region 31 in step S331.

As a result of the determination, if it is determined that the startingcoordinate (X1, Y1) is within the predetermined region 31 (Yes in stepS331), the process proceeds to step S332, and if not within thepredetermined region 31 (No in step S331), the process proceeds to stepS334.

Next, in step S332, the control unit 100 determines whether or not theending coordinate (X2, Y2) obtained in step S29 shown in FIG. 5 iswithin a predetermined region at a lower center of the display unit.

FIG. 11 is a view showing the predetermined region at the lower centerof the display unit. As shown in FIG. 11, when the right index fingerFfr is lifted at the lower center of the display unit after the uppercenter of the display unit is pressed with the right index finger Ffr asshown in FIG. 10, a coordinate corresponding to the position at whichthe finger has been lifted is stored as the ending coordinate (X2, Y2).In this case, the right index finger Ffr is lifted (separated from thedisplay unit) within a predetermined region 32 in the lower center, andtherefore it is determined that the ending coordinate (X2, Y2) is withinthe predetermined region 32 in step S332.

As a result of the determination, if it is determined that the endingcoordinate (X2, Y2) is within the predetermined region 32 (Yes in stepS332), the process proceeds to step S333, and if not within thepredetermined region 32 (No in step S332), the book-reading processingis terminated, and the processing shown in FIG. 8 (and further theprocessing shown in FIG. 4) is resumed.

Subsequently, in step S333 (bookmark application processing), thebookmark application processing as shown in FIG. 12 is performed, as itis determined that the user has inputted a gesture, in steps S331 andS332, as a command for executing the bookmark processing.

FIG. 12 is a view showing an example of the screen shown when performingthe bookmark application processing. As shown in FIG. 12, a plurality ofalphabets are presented in a bookmark section 333 displayed at an upperportion of the screen from the left display unit 14 a to the rightdisplay unit 14 b, and each alphabet corresponds to one bookmark.Specifically, when a position indicated by “N” in the bookmark section333 is touched with the right index finger Ffr, page numbers of imagescorresponding to book data currently displayed in the left display unit14 a and the right display unit 14 b are stored in the storage unit 120in association with a bookmark “N”. The information relating to thebookmark remains stored even after the operation of the device isterminated. If it is desired to browse the images of these pages againafter a certain time period, by selecting the bookmark “N” in a bookmarkcalling screen similar to FIG. 12 but not shown in the drawing, thedesired images can be immediately called without the user inputting suchas the page number of the image corresponding to the book data.

As described above, by touching the screen with the right index fingerfrom (the predetermined region 31 in) the upper center of the screen to(the predetermined region 32 in) the lower center of the screen, thepages are stored as if a bookmark is placed between the two displayedpages. Therefore, the user can carry out the bookmark applicationprocessing in a simple manner by an intuitive operation.

Next, in step S334 shown in FIG. 9, the control unit 100 determineswhether or not the starting coordinate (X1, Y1) and the endingcoordinate (X2, Y2) are in different screens. Specifically, it isdetermined if one of the starting coordinate (X1, Y1) and the endingcoordinate (X2, Y2) is a coordinate within the left display unit 14 aand the other of the starting coordinate (X1, Y1) and the endingcoordinate (X2, Y2) is a coordinate within the right display unit 14 b.This determines whether or not a gesture of the right index fingermoving from the left display unit 14 a to the right display unit 14 band vice versa, for example, has been inputted. In the followingdescription, this is specifically described with reference to FIG. 13and FIG. 14.

FIG. 13 is a view showing an example in which the gesture of the rightindex finger moving from a left screen to a right screen is inputted,and FIG. 14 is a view showing an example in which the gesture of theright index finger moving from the right screen to the left screen isinputted.

Referring to FIG. 13, as the left display unit 14 a is pressed with aright index finger Ffr′ (at a time point in the past), a coordinatecorresponding to the pressed position is stored as the startingcoordinate (X1, Y1). Then, as the right index finger Ffr is lifted fromthe right display unit 14 b (at a current time point), a coordinatecorresponding to the lifted position is stored as the ending coordinate(X2, Y2). Further, in FIG. 14, the gesture of an opposite direction ismade. Accordingly, in this case, the starting coordinate and the endingcoordinate that are stored are opposite of the case shown in FIG. 13.

In step S334, if it is determined that the gesture as a command forflipping a page has been made (Yes in step S334), the process proceedsto step S336, and a direction of the page flipping is determined. Forexample, if the starting coordinate is smaller than the endingcoordinate (X1<X2), it is recognized that a gesture for flipping fromleft to right has been made, and the direction of the page flipping isdetermined to be a forward direction. Then, the process proceeds to stepS337. Further, if it is determined that the gesture for flipping a pagehas not been made (No in step S334), the process proceeds to step S335.

Next, in step S335, the control unit 100 determines whether or not thestarting coordinate (X1, Y1) and the ending coordinate (X2, Y2) are bothat a predetermined position for instructing successive page flipping.This determines whether or not a (stationary) gesture of a right thumbpressing near a right end portion of the right display unit 14 b for apredetermined time, for example, has been inputted. In the followingdescription, this is specifically described with reference to FIG. 15.

FIG. 15 is a view showing an example in which the gesture of the rightthumb pressing near the right end portion in the right screen isinputted. Referring to FIG. 15, as a right thumb Ftr presses apredetermined region 33 b for instructing successive page flipping thatis provided near the right end portion of the right display unit 14 b, acoordinate corresponding to the pressed position is stored as thestarting coordinate (X1, Y1). Then, if the predetermined region 33 b iskept being pressed as it is after a predetermined time elapses, thecoordinate corresponding to this position is stored as the endingcoordinate (X2, Y2) even if the right thumb Ftr is not lifted.

Specifically, as previously described according to step S27 shown inFIG. 5, if the inputted coordinate does not change (over thepredetermined range), the short time (for example, on the order of 1second) is set as the predetermined time, and the processing insubsequent step S29 is performed upon elapsing of this time, and theending coordinate (X2, Y2) is stored. Therefore, if the predeterminedregion 33 b is kept being pressed, the processing including theprocessing in step S335 and processing in step S337 that will bedescribed later are repeated every predetermined short time (forexample, 1 second), and the processing for flipping a page in theforward direction is repeated. Further, if the predetermined region 33 ashown in FIG. 15 is kept being pressed with the right index finger Ffr,for example, the processing for flipping a page in an opposite directionis repeated.

In step S335, if it is determined that the stationary gesture as acommand for successively flipping a page has been inputted (Yes in stepS335), the process proceeds to step S336, and a direction of the pageflipping is determined. For example, if the predetermined region 33 b inthe right display unit 14 b is pressed, it is recognized that astationary gesture for flipping a page from left to right has been made,and the direction of the page flipping is determined to be the forwarddirection. Then, the process proceeds to step S337. Further, if it isdetermined that the gesture for successively flipping a page has notbeen made (No in step S335), it is determined that a command relating tothe book-reading processing has not finally been inputted (including thecase in which the starting coordinate and the ending coordinate areinitialized). Therefore, the book-reading processing is terminated, andthe processing shown in FIG. 8 (and further the processing shown in FIG.4) is resumed.

Subsequently, in step S337, according to the direction of the pageflipping determined in step S335, images corresponding to pages ofnumbers incremented or decremented by two from the current pages arerespectively displayed in the left display unit 14 a and the rightdisplay unit 14 b. Then, the book-reading processing is terminated, andthe processing shown in FIG. 8 (and further the processing shown in FIG.4) is resumed.

It should be noted that the number of pages incremented or decrementedaccording to the direction of the page flipping is generally two, but itis possible to employ a configuration in which this number is set by theuser as needed. Further, for example, it is possible to employ aconfiguration in which the number of pages to be flipped changes asneeded, according to such as a distance or a positional relation betweenthe starting coordinate and the ending coordinate, or elapsed time fromthe input of the starting coordinate to the input of the endingcoordinate. Examples can include a configuration in which the number ofpages to be flipped increases as the distance between the startingcoordinate and the ending coordinate becomes larger, or the number ofpages to be flipped increases as the elapsed time is shorter (that is,as the gesture is made more quickly). Next, the window processing in thewindow mode is described in detail.

4.3 Window Processing Operation in Window Mode

FIG. 16 is a flowchart showing a detailed flow of the window processing(step S35). In step S351 shown in FIG. 16, the control unit 100determines, based on the starting coordinate (X1, Y1) and the endingcoordinate (X2, Y2) that have been obtained in step S25 shown in FIG. 5,whether or not a gesture that has been inputted by the user matches anyof gestures corresponding to commands for executing previouslydetermined window processing. The gestures are previously set inassociation with commands for changing a form of the window from thecurrent state, more particularly, increasing or decreasing the dividingnumber of the window, or changing a form of divisions of the window (inthis case, dividing vertically or horizontally). A relation between thegestures and changing forms of the window is described with reference toFIG. 17.

FIG. 17 is a view showing the relation between changing forms of thewindow and previously determined gestures. Referring to FIG. 17, fourforms of the window are set: (seemingly) one screen constituted by theleft display unit 14 a and the right display unit 14 b, two verticalscreens divided at the center right and left (vertically divided), twohorizontal screens divided at the center top and bottom (horizontallydivided), and four screens divided top-bottom and right-left.Transitions from one of these window forms to another are shown byarrows, and gestures set as commands for instructing the transitions(changes) of the window form are briefly illustrated near thecorresponding arrows.

These gestures include gestures A1-A3 having the upper center of thescreen as the starting coordinate, a gesture B1 having the lower centerof the screen as the starting coordinate, gestures C1 and C2 having theleft end center portion of the screen as the starting coordinate,gestures D1 and D2 having the right end center portion of the screen asthe starting coordinate, gestures E1 and E2 having a portion along rightand left directions from the center portion of the screen as thestarting coordinate, and gestures F1-F4 having the center portion of thescreen as the starting coordinate. FIG. 17 also shows a gesture A3′having the center portion of the screen as the starting coordinate, theleft end center portion of the screen as the intermediate pointcoordinate, and the right end center portion of the screen as the endingcoordinate. However, for convenience of explanation here, therecognition of the intermediate coordinate is omitted, and the gestureA3′ is considered to be the same as the gesture A3. It should beappreciated that in order to correctly recognize the gesture A3′, it ispossible to successively detect the intermediate point coordinatesduring a time period from bringing down to lifting up of the right indexfinger Ffr and to determine whether or not any of the intermediate pointcoordinates is included within the predetermined region of the left endcenter portion of the screen.

It should be noted that while these gestures are a practical exampleconceived so that the user can intuitively learn, these are mereillustrations and any practical example can be employed as long as afrom is easy to recognize or remember for the user. Further, althoughthe numbers of divisions of the window is 1 (undivided), 2, or 4, forconvenience of explanation here, but the number of division is notparticularly limited. In addition, the window can be divided in any fromof division such as horizontally dividing into four or dividing in awell-known special form.

Here, in order to change the window from one screen to two verticalscreens, the gesture A1 can be performed taking the upper center of thescreen as the starting coordinate and the lower center of the screen asthe ending coordinate. It should be noted that although this issubstantially the same as the gesture for executing the bookmarkprocessing, as described according to step S31 of FIG. 8, when (theoperation mode selected by the user is) in the reading mode, the gestureis recognized as the gesture for executing the bookmark processing asthe book-reading processing, and when not in the reading mode, thegesture is recognized as the gesture for changing the form of the windowas the window processing.

Further, in order to change the window from two vertical screens to twohorizontal screens, the gestures A2 and A3 can be performed as shown inFIG. 17, and in order to change the window from two horizontal screensto one screen, the gestures E1 and E2 can be performed as shown in FIG.17.

Moreover, for example, in order to change the window from four screensto one screen, one of the gesture F3 taking the center portion of thescreen as the starting coordinate and a portion near an upper left endof the screen as the ending coordinate, and the gesture F4 taking thecenter portion of the screen as the starting coordinate and the sameposition as the ending coordinate by keeping pressing this position forthe predetermined time can be performed. It should be noted here that,the gesture F3 includes a gesture taking the center portion of thescreen as the starting coordinate and a portion near one of an upperright end, a lower right end, and a lower left end of the screen as theending coordinate. Further, the other gestures for changing the windowform are apparent by reference to FIG. 17, and a detailed explanation isomitted.

As described above, in step S351 shown in FIG. 16, the control unit 100determines whether or not the inputted gesture matches any of thegestures shown in FIG. 17, and if the gesture does not match as a resultof the determination (No in step S351), a gesture corresponding to acommand for executing the window processing has not been inputted, thewindow processing is terminated and the processing shown in FIG. 8 (andfurther the processing shown in FIG. 4) is resumed. Alternatively, ifthe gesture matches (Yes in step S351), the process proceeds to stepS353.

In step S353, the control unit 100 determines whether or not commandprocessing corresponding to the inputted gesture is executable in thecurrent window form. As a result of the determination, if it isdetermined to be unexecutable (No in step S353), for example, if thegesture F1 is inputted when the screen is one screen, there is no windowform to be transited (changed), and corresponding processing command(here, changing processing from two vertical screens to one screen, orfrom two horizontal screens to four screens) is unexecutable, and theprocessing is terminated. If it is determined to be executable (Yes instep S353), the form of the window is caused to change according tocommand processing corresponding to the gesture (step S355), and thenthe window processing is terminated and the processing shown in FIG. 8(and further the processing shown in FIG. 4) is resumed.

5. Effects

As described above, according to this embodiment, a portable displaydevice that is sufficiently small to be held with one hand enters astate in which a gesture can be accepted when a fixed coordinateposition near a central portion between the two screens is pressed withthe thumb of one hand holding the device, and a command for executingthe book-reading processing or the window processing is accepted by apredetermined gesture inputted typically made by the finger of the otherhand. Therefore, holding the two-screen portable display deviceconveniently and naturally causes the device to enter the commandaccepting state to allow gesture recognition, and when the portabledisplay device is not held, the device enters the command non-acceptingstate to prevent a command from being falsely executed due to anunintended contact and such to the display screen. In this manner, theportable display device according to this embodiment can provide aninterface for input operations suitable for a two-screen display screen.

6. Modified Examples

According to the embodiment, various gestures corresponding to commandsfor executing the bookmark application processing, the page flippingprocessing, the window processing, and such are described, but thesegestures are mere examples. Any gestures that can be recognized as achanging form of the two or more associated coordinates bychronologically associating the inputted coordinates can be included, orprocessing command previously stored so as to be associated with thesegestures may include any processing to be executed in the potabledisplay device, as long as the device enters the command non-acceptingstate when the inputted coordinates do not include coordinates withinthe fixed coordinate area, and enters the command accepting state whenthe inputted coordinates include coordinates within the fixed coordinatearea. For example, it is possible to realize an operation for executinga command for enlarging an image displayed in the right display unit 14b by making a gesture of pressing the fixed coordinate position near thecentral portion between the two screens with the thumb of one handholding the device, and of placing the thumb and the index finger of theother hand on the right display unit 14 b and spreading the fingers, orby making a gesture of moving the hand finger of the other hand fromlower left to upper right of the right display unit 14 b. Alternatively,to the contrary, it is possible to realize an operation for executing acommand for reducing an image displayed in the right display unit 14 bby making a gesture of pressing the fixed coordinate position near thecentral portion between the two screens with the thumb of one handholding the device, and of placing the thumb and the index finger of theother hand on the right display unit 14 b and contracting the fingers,or by making a gesture of moving the hand finger of the other hand fromupper right to lower left of the right display unit 14 b.

According to this embodiment, it is described that the fixed coordinateposition near the central portion between the two screens is pressedwith the thumb of one hand holding the portable display device. This isbecause it is generally configured such that a portion near the centralportion between the two screens can be held most easily. However, thereis a case in which a position that the user thinks can be held mosteasily is different from this position, and or a case in which attachingaccessories to the device changes the position that generally can beheld most easily. Therefore, the fixed coordinate position can bealtered to a predetermined position that is distant from the portionnear the central portion between the two screens, for example, to suchas a portion near a central portion of the left end of the left displayunit 14 a.

According to this embodiment, the recognition processing (S3) isperformed after the input processing (S2) ends. However, such processingsteps (including other processing steps) are mere examples forconvenience of explanation, and the processing can be performedintegrally or well-known processing steps can be employed such asperforming event-driven type processing.

According to this embodiment, the types of the gestures and the commandsassociated with the gestures (processing operations) are previously andfixedly stored. However, the association can be freely set by the useror by an application.

According to this embodiment, the description is given taking theexample of the portable display device having two screens. However, thedevice can have three or more screens as long as the above describedgestures can be made for two adjacent screens of the three or morescreens.

According to this embodiment, the ending coordinate is the position atthe moment when the finger is lifted. However, the ending coordinate canbe determined at a time point when the finger is separated from thedisplay unit for a predetermined time. By doing so, when a gesture madeover two screens is inputted in a state in which the left display unit14 a and the right display unit 14 b are slightly valley-folded, andwhen the finger is lifted for a moment at the portion near the centralportion, for example, a position where the finger is last lifted (forover the predetermined time) is taken as the ending coordinate, and itis possible to recognize the gesture as a successive gesture as a whole.

According to this embodiment, the gesture is recognized based on thestarting coordinate and the ending coordinate (as well as the elapsedtime). However, it is possible to use various well-known methods forrecognizing the gestures. For example, the gesture can be recognized bywell-known pattern recognition, by a predetermined vector operation, orby determining which one of the gestures corresponds based on a changein a group of the related (successive) coordinates stored every unittime.

According to this embodiment, there is no gap between the left displayunit 14 a and the right display unit 14 b (seamless), but a certaindegree of a gap may present. However, it is necessary that the gap canbe pressed typically with the thumb, and the pressed state can bedetected, and it is desirable that there be little unevenness so that agesture over right and left screens can be made as described above.

According to this embodiment, the gesture recognition described above isimplemented in the portable information terminal. However, it ispossible to implement the gesture recognition in any well-known device,such as a mobile telephone, an electronic personal organizer, anelectronic dictionary, an electronic book reader, or a mobile Internetterminal, as long as the portable display device can be held by theuser.

The present invention is applied, for example, to a portable displaydevice having a touch panel, and suitable for a portable display devicehaving a two-screen display unit and a touch panel capable of detectinga coordinate of a position approached or pressed on the display unit.

1. A portable display device configured to be held by a user, the devicecomprising: a display unit configured to display images respectively intwo adjacent screens; an input unit configured to obtain two or morecoordinates of positions on the display unit, the positions beingsubjected to one of approach, contact, and press by the user; and agesture recognizing unit configured to recognize and execute apreviously stored processing command corresponding to a changing form oftwo or more related coordinates by chronologically associating thecoordinates of the positions subjected to one of approach, contact, andpress on the display unit and obtained by the input unit, wherein thegesture recognizing unit enters: a non-accepting state in whichrecognition and execution of the processing command are unallowed whenthe coordinates obtained by the input unit do not include apredetermined fixed coordinate, and an accepting state in whichrecognition of the processing command is allowed when the coordinatesobtained by the input unit include the fixed coordinate.
 2. The portabledisplay device according to claim 1, wherein the gesture recognizingunit executes the processing command based on the fixed coordinate thathave been previously determined, the fixed coordinate being at or near aportion at which the two screens are adjacent.
 3. The portable displaydevice according to claim 1, wherein the input unit obtains, as thefixed coordinate, a coordinate of a position on the display unit that iscorresponding to a position of one hand finger of the user holding theportable display device and that is corresponding to a position suitablefor being held by the one hand finger of the user, and obtains, as thetwo or more related coordinates, coordinates of positions on the displayunit that are corresponding to a position of the other hand finger ofthe user.
 4. The portable display device according to claim 1, whereinthe display unit is foldable such that the two screens face toward eachother taking a portion at or near a side where the two screens areadjacent as a valley-folding line, and the gesture recognizing unitexecutes the processing command based on the fixed coordinate includinga coordinate on or near the valley-folding line.
 5. The portable displaydevice according to claim 1, wherein when the two or more relatedcoordinates represent a changing form in which the coordinates move fromone to the other of the two screens, the gesture recognizing unitrecognizes and executes a predetermined page forwarding commandcorresponding to the changing form, and the display unit displays twoimages respectively associated with predetermined pages in a two-pagecenter spread manner in the two screens, and displays images associatedwith pages incremented or decremented by a predetermined numberaccording to the changing form when the page forwarding command isexecuted by the gesture recognizing unit.
 6. The portable display deviceaccording to claim 1, wherein when the coordinates obtained by the inputunit include previously determined coordinates successively for apredetermined time period, the gesture recognizing unit recognizes andexecutes a predetermined page forwarding command during this period, thepreviously determined coordinates being positioned near an outercircumference of the two screens and distant from the fixed coordinateby a predetermined distance, and the display unit displays two imagesrespectively associated with predetermined pages in a two-page centerspread manner in the two screens, and displays images associated withpages incremented or decremented by a predetermined number during theperiod at a predetermined time interval when the page forwarding commandis recognized by the gesture recognizing unit.
 7. The portable displaydevice according to claim 1, wherein the display unit displays twoimages respectively associated with predetermined pages in a two-pagecenter spread manner in the two screens, and when the two or morerelated coordinates represent a changing form in which the two or morerelated coordinates move from a portion near an end of the screen to aportion near a center of the screen at or near a portion at which thetwo screens are adjacent, the gesture recognizing unit recognizes apredetermined bookmark assigning command, and stores pages correspondingto the images displayed in the display unit.
 8. The portable displaydevice according to claim 1, wherein when the two or more relatedcoordinates represent a changing form in which the two or more relatedcoordinates move from a portion near one side to a portion near anotherside out of four outer sides of at least one square display window inthe two screens, the gesture recognizing unit recognizes and executes apredetermined window form changing command corresponding to the changingform, and when the window form changing command is executed by thegesture recognizing unit, the display unit changes one of the number anda form of divisions of the window according to the changing form, anddisplays the images in the window in the form after the change.
 9. Theportable display device according to claim 1, wherein the gesturerecognizing unit enters: a non-accepting state in which recognition andexecution of the processing command are unallowed when the coordinatesobtained by the input unit do not include a predetermined number offixed coordinates out of a plurality of predetermined fixed coordinatesat or near a portion at which the two screens are adjacent, and anaccepting state in which recognition of the processing command isallowed when the coordinates obtained by the input unit include thepredetermined number of fixed coordinates.
 10. The portable displaydevice according to claim 1, wherein during a time period in which thegesture recognizing unit is in the non-accepting state, the input unitperforms at least one of an operation of limiting a range of coordinatesto be obtained on the display unit to the fixed coordinate or near thefixed coordinate, and an operation of setting a time interval at whichthe coordinates on the display unit are to be obtained to be longer thana time period during which the gesture recognizing unit is in theaccepting state, thereby obtaining the coordinates.
 11. A method ofcontrolling a portable display device configured to be held by a userand having a display unit for displaying an image over two adjacentscreens, the method comprising: an input step of obtaining two or morecoordinates of positions on the display unit, the positions beingsubjected to one of approach, contact, and press by the user; and agesture recognizing step of recognizing and executing a previouslystored processing command corresponding to a changing form of two ormore related coordinates by chronologically associating the coordinatesof the positions subjected to one of approach, contact, and press on thedisplay unit and obtained in the input step, wherein the gesturerecognizing step is in: a non-accepting state in which recognition andexecution of the processing command are unallowed when the coordinatesobtained in the input step do not include a previously determined fixedcoordinate at or near a portion at which the two screens are adjacent,and an accepting state in which recognition of the processing command isallowed when the coordinates obtained in the input step include thefixed coordinate.
 12. A program for causing a portable display devicehaving a display unit for displaying an image over two adjacent screensand configured to be held by a user to execute: an input step ofobtaining two or more coordinates of positions on the display unit, thepositions being subjected to one of approach, contact, and press by theuser; and a gesture recognizing step of recognizing and executing apreviously stored processing command corresponding to a changing form oftwo or more related coordinates by chronologically associating thecoordinates of the positions subjected to one of approach, contact, andpress on the display unit and obtained in the input step, wherein thegesture recognizing step is in: a non-accepting state in whichrecognition and execution of the processing command are unallowed whenthe coordinates obtained in the input step do not include a previouslydetermined fixed coordinate at or near a portion at which the twoscreens are adjacent, and an accepting state in which recognition of theprocessing command is allowed when the coordinates obtained in the inputstep include the fixed coordinate.
 13. A computer-readablenon-transitory recording medium having a program recorded therein, theprogram causing a portable display device having a display unit fordisplaying an image over two adjacent screens and configured to be heldby a user to execute: an input step of obtaining two or more coordinatesof positions on the display unit, the positions being subjected to oneof approach, contact, and press by the user; and a gesture recognizingstep of recognizing and executing a previously stored processing commandcorresponding to a changing form of two or more related coordinates bychronologically associating the coordinates of the positions subjectedto one of approach, contact, and press on the display unit and obtainedin the input step, wherein the gesture recognizing step is in: anon-accepting state in which recognition and execution of the processingcommand are unallowed when the coordinates obtained in the input step donot include a previously determined fixed coordinate at or near aportion at which the two screens are adjacent, and an accepting state inwhich recognition of the processing command is allowed when thecoordinates obtained in the input step include the fixed coordinate.